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VV&C - Importance of Communication 
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W&C - Defining Terms for M&S ife* 

e 


•Verification: “...computational model 
accurately represents the underlying 
mathematical model...” 

• Validation: “...determining the degree to 
which a model ... is an accurate 
representation of the real world...” 

• Credibility: “the quality to elicit belief or 
trust” 


NASA-STD-7009, ‘‘Standard for Models and Simulations”, 2013. 
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W&C - Credibility Is Key 




• M ASA 


• Credibility begins before development 

• Includes the end users 

• Is an ongoing assessment 
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IMM Project VV&C History (April 2008 ) 


Scenario Definition (DRM) 


iMED Database 
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Probabilistic Risk Analysis / Monte Carlo Simulation 



Medical Events 

• Crew Impairment 

• Loss of Crew Life 

• Evacuation 

• Resources Consumed 
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IMM Project VV&C History (April 2008 ) 
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Probabilistic Risk Analysis / Monte Carlo Simulation 



IMM Output Results 


Medical Events 
Crew Impairment 
Loss of Crew Life 
Evacuation 
Resources Consumed 



National Aeronautics and Space Administration 


7 




IMM Project VV&C History (April 2008 ) 
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I MM Project VV&C History (April 2008 ) i&a 


r 


Scenario Definition (DRM) iMED Database 
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IMM Project VV&C History (April 2008 ) 
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Scenario Definition (DRM) 


iMED Database 






Probabilistic Risk Analysis / Monte Carlo Simulation 



IMM Output Results 
Individual outputs 
DRM Results 
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NASA-Standard-7009 (July 2008 ) 


nVV- 


Programmatics (7/49) 

Simulations and 
analyses (10/49) 

Recommended practices: 
identification and use 
(1/49) 

Credibility assessment of 
model and simulation 
(M&S) results (3/49) 


Models (13/49) 


V&V and uncertainty 
quantification (9/49) 


Training (3/49) 


Reporting results to 
decision makers (3/49) 
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IMM 7009 Compliance Summary (IMM v3) 


49 Requirements 

Compliant Compliance Incomplete Not Applicable 
42 5 2 
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9 Compliance Summary (IMM v3) 


49 Requirements 


Compliant 

Compliance Incomplete 

Not Applicable 
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Simulations & analyses (10/10 I V&V and uncertainty (8/9 



Recommended practices (0/1 ) 
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9 Compliance Summary (IMM v3) NASA 


49 Requirements 


Compliant 

Compliance Incomplete 

Not Applicable 
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Programmatics (6/6) 


Models (11/12) 


Simulations & analyses (10/10) V&V and uncertainty (8/9 




Recommended practices (0/1) 



Training (1/3) 


Credibility ass 



Reporting results (3/3) 
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W&C IMM 4 
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IMM 4.0 
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W&C IMM 4.0 
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W&C IMM 4.0 
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IMM 4.0 Comparison to RWS 


nVV- 


Scenario Definition (DRM) 


iMED Database 






Probabilistic Risk Analysis / Monte Carlo Simulation 



IMM Outputs 
Individual outputs 
DRM Results 

Real world system (RWS) I 
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4.0 Comparison to RWS 


* f m 

- • NASA 


iMED (current) 


Validation data 


(STS thru STS-114 


STS-115 - STS-135 


ISS thru Exp13 


Post ISS Exp13 
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Example RWS Results (Quantitative) 


•Trends and distributions 
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NASA 


^kstrom, P-A, Eikos: A Simulation Toolbox for Sensitivity Analysis, Uppsala Universitet, 2005 
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External Review- Future 


Scenario Definition (DRM) 


iMED Database 
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Probabilistic Risk Analysis / Monte Carlo Simulation 



Medical Events 

• Crew Impairment 

• Loss of Crew Life 

• Evacuation 
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External Review: Software and Code nW 
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External Review: Input Process 
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I MM 4.0 End-to-End External Review i&a 
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• 7009 Technical Review 

-Evidence 
-Processes 
-Identify limitations 
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4.0 End-to-End External Review 
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IMM 4.0 Transition to Ops 
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IMM Project W&C Conclusions 

’• •* • 


• Current operational IMM 

-42/47 compliant 

-5/47 compliance incomplete 

• IMM 4.0 

-Functionality results JPR 7150.2 
-RWS validation -> 7009 Technical review 
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IMM PROJECT VERIFICATION, 

VALIDATION, AND 
CREDIBILITY 

Thank you! 

\ i I , j - Questions? 
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Back up Slides 



I MM Assessment Report 


Validation status for 
IMM 


IMM V&V Matrix: The following bar chart shows the IMM version three MATLAB status 
of internal (black bars) and external (gray bars) IMM validation for each IMM element listed. 



Compliance with NASA 

Standard-7009 

requirements 


IMM Compliance Matrix: The IMM has no waivers to any of the requirements in the 
NASA-Standard-7009 for Models and Simulations (4); however there are certain requirements 
that are not applicable to the IMM and other requirements where IMM compliance has not yet 
been fully attained. 


Compliant Not yet compliant Not applicable 
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Current NASA- 
Standard-7009 credibility 
assessment score 


IMM Credibility Matrix: The following table shows the overall IMM version three 
MATLAB credibility assessment score (CAS), individual factor CAS, customer-specified 
sufficiency thresholds, and weighted scores. Legend depicts CAS score designated range and 
color assignments from NASA-Standard-7009. Review subfactor weights for bolded factors is 


0.3. 


Legend 


CAS Score > Threshold 


Threshold > CAS Score > Threshold-0.5 


Threshold-0.5 > CAS Score > Threshold-1.0 


CAS Score < Threshold - 1.0 


Overall score indicating sufficiency thresholds metl 

3.075 f 


Factor 

Sufficiency 

Thresholds 

Factor 

Score 

Weighted 

Scores 

Overall 

Score 

Verification 

4 

3 

0.225 


Validation 

3 

2.3 

0.2925 


Input Pedigree 

3 


0.28 


Results Uncertainty 

3 

3 

0.6 

2.50 

Results Robustness 

3 

3 

0.3975 


Use History 

3 

2 

0.375 


M&S Management 

2 

2 

0.1 


People Qualification 

4 

4 

0.2 
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